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(57) Abstract: Methods and apparatus for managing, finding and displaying objects such as digital images. Objects are tagged 
(••associated") with descriptive textual and numeric data ("metadata"), and stored in a relational database from which they can be 
selected, sorted, and found. Tags can be defined by name, tag type, and associated attributes. Objects can be tagged by dropping a 
tag onto the object, or relating a database record for the tag to a database record for the object. Tagged objects can be searched for and 
displayed according to the degree to which their metadata matches the search criteria. Visual cues can indicate whether displayed 
objects match all, some but not all, or none of the search criteria. Database object distributions can be displayed as histograms or 
scatter plots, including timelines, calendars or maps. Object distributions can be used to search for objects or to limit search results 
for a previous search. 
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DIGITAL MEDIA MANAGEMENT APPARATUS AND METHODS 



Background 

With the advent of digital photography and the world- wide-web, there has 
been an exponential growth in the creation and storage of digital photographic images. 
5 As the number of digital photographs taken and stored has grown, so too has the need 
for a convenient method of archiving, cataloguing, searching, and retrieving them. 
Modem metiiods of archiving and storing digital images typically require users to 
remember large amounts of information merely to locate photos that are of particular 
interest to them. For example, many users currently store their digital images in the 
10 hierarchical, directory-based file system structure that is native to personal computers. 
To find particular photos stored in such a hierarchical directory tree or structure, users 
must know the fiill pathname to the directory in which their photographs are stored. 

There are other disadvantages to storing digital photographs in a hierarchical, 
directory-based file system. For example, cataloguing and storing groups of photos by 

15 categories such as vacation photos or wedding photos requires creating different 
directories for each of the desired categories. This fiulher increases the amoimt of 
information that must be remembered in order to locate desired photos. In addition, in 
ordCT to store photos in two or more overlapping categories, such as photos that 
include your favorite aunt and photos from your cousin's wedding, users must either 

20 store duplicate photographs, or master the concepts of directory trees and file pointers. 
While these are not diflBcult concepts for sophisticated computer users, they can be 
troublesome for less sophisticated users, thereby limiting the usefiil ways these users 
can store and retrieve digital photographs and photographic information. 

Summary 

25 The present invention relates to methods and apparatus for storing, 

cataloguing, managing, organizing, finding and displaying objects such as digital 
images. The invention includes methods for associating ("ta^jng") fields of text and 
numeric data ("metadata") wifii mdividual objects such as images or photos, storing 
the objects and associated metadata as records in a relational database, and selecting, 

30 sorting, organizing and finding the objects based on their tagged metadata content 
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Default metadata tags can be specified, and new metadata tags can be defined 
and created through a tag editor by naming the tag, selecting its tag type, optionally 
selecting a graphical icon that represents the tag, and filling in any remaining fields or 
attributes that are unique to and define the tag type. Tags can be readily associated 
5 with an object by adding a record containing the tag information or metadata to a 
database, and relating the tagged metadata record to a database record containing the 
object or a pointer to the object Tags can also be graphically associated with an 
object by, for example, dragging and dropping a graphical icon representing the tag 
onto a graphical representation of the object. In the latter case, database records 
10 containing the tag metadata are automatically created and related to the database 
record containmg the target object or a pointer to the target object 

Once objects have been tagged wilb metadata, they can be searched for 
according to one or more tagged search criteria. When the objects to be search for are 
photos, these search criteria can include, but are not limited to, the date and time tiie 
1 5 photos were taken, textual information that is associated with the photos such as the 
names of tiie people who are in the photos or the places or events where the photos 
were taken, designations of the photos as favorite photos, and designation of the 
photos as photos that have been printed, shared with others, or archived on a certain 
date. 

20 When a database is searched for objects that match one or more tagged search 

criteria, the matching objects can be viewed or arranged according to the degree to 
which they have associated metadata that matches the search criteria, hi particular, 
objects tiiat match all of the search criteria can be displayed first, followed by objects 
that match one or more of the search criteria, and finally by objects that match none of 

25 the search criteria. Objects in the different match groups can be differentiated from 
one another in the display area by visual cues, such as being displayed in front of 
different background colors or patterns. Thus, objects matching all of the search 
criteria can be displayed in front of a white background, while objects matching some 
of the search criteria can be displayed in &ont of a blue background, and objects 
30 matching none of the search criteria can be displayed in front of a gray background. 
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The distribution of the objects stored in titte database can be displayed as a 
histogram along a timeline. Time bands can be set along the timeline to indicate a 
time period that can be used to search for matching objects in the database, or to limit 
the search results for a given tag search to objects having temporal metadata within 
5 the indicated time period. When the timeline is used to limit the search results for a 
tajg search, the timeline displays not only the temporal distribution of all objects in the 
database over the indicated time p^od, but also the temporal distribution of all 
objects in the database matching the specified tag search criteria over the indicated 
time period. 

10 In addition to timelines, the temporal distribution of objects in the database 

can be represented in a calendar view such that the days of tiie calendar indicate the 
number of objects having metadata associated with a given day of the week in a given 
week of die month. The calendar view can also be used to limit the search results for 
a tag search, in >^diich case the calendar view will indicate all ofHie days of the month 

1 5 associated with objects that match all of the tagged search criteria, matoh some of the 
tagged search criteria, and match none of the tagged search criteria. 

The details of one or more embodiments of the invention are set forth in the 
accompanying drawings and the description below. Other objects, features, and 
advantages of the invention will be apparent from the description and drawings, and 
20 the claims. 

Brief Description of Drawings 

Figs. 1 illustrates one embodiment of a usct interface for a computer program product 
in accordance widi the present invention. 

Fig. 2 illustrates an image displayed with its associated metadata, including its tags, in 
25 accordance with tiie present invention. 

Fig- 3 illustrates a timeline view of the data in accordance with the present invention. 

Fig. 4 illustrates a calendar view of the data in accordance with the present invention. 

Fig. 5 illustrates a map view of the data in accordance with the present invention. 
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Fig. 6 illustrates the display of diflferent media types that are stored in accordance with 
the present invention. 

Detailed Description 

The present invention provides a method for users to organize and find digital 
images and photos by tagging them. Before being tagged, photos must be imported 
into a database where photographic metadata or information about the photos can be 
stored. While entire photos can be stored in the database, it is generally more efiBcient 
to store pointers to photos in the database rather than the photos themselves. Photos 
can be imported into the database from any of a number of devices or sources 
including, but not limited to, a digital camera, a flash memory device, a hard disk 
drive, a floppy drive, a CD-ROM, or a networked computer or file server. Once 
imported into the database, the photos can be tagged with one or more objects 
containing metadata that identifies the unique or important properties of the photo 
such as when or where the photo was taken, or who or what is the subject of the 
photo. 

As shown in Figs. 1, in one embodiment tags 350 can be applied to photos by 
dragging and dropping graphical icons representing the tags onto one or more photos 
1-4 that are displayed in an image area 100. When a tag is dropped onto a photo, the 
database record that contains a pointer to the photo is updated to contain or point to 
metadata that is associated with the tag that has been dropped onto the photo. This 
metadata can include when the photo was taken, where it was taken, the nature of the 
event at which it was taken, the subject of the photo, and whether the user considers 
the photo one of his or her favorites. Once tagged, photos with specific tags or 
combinations of tags can be readily found in the database by searching the database 
for all records that contain the same metadata as the metadata that is associated with 
the one or more search tags. 

Tags, and the metadata they contain, can be created and modified in a tag 
editor. The tag editor allows a user to specify a tag name and tag type, and to enter 
metadata in the form of tag attributes that can be stored in tags of the specified tag 
type. For convenience, tags can be divided into one or more tag categories. For 
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example, in one embodiment tags are divided into people, events, places and 
miscellaneous tag categories. in the different tag categories generally have 
different tag attributes to distinguish between tiiemselves and tags in other tag 
categories. In general, a tag's attributes do not need to be filled in to associate a tag 
5 with a photo. The tag itself is a form of metadata that can be associated with the 
photo, regardless of whether the tag*s possible attributes are also associated with the 
photo. However, when a tag's attributes are completely or partially filled in, more 
metadata is associated with the tagged photo, thereby making the photo easier to 
search for and find. 

1 0 The people tag category includes default tag types for family and fiiends, and 

can be customized to include other groups of people such as business associates, 
classmates, co-workers, and neighbors, and particular individuals such as a spouse, 
dau^ter, or fiiend. Tags in the people category can contain attributes such as a 
person's name, sex, birthdate, anniversary, postal and/or email address(es), phone 

1 5 number(s), a sharing profile specifying which if any pictures can be shared with the 
people associated with the tag, and the relationships between the people associated 
with tibie tag and other tagged individuals. 

The events tag category includes default tag types for parties and vacations, 
and can be ctistomized to include tag types for particular types of events such as 

20 concerts, plays, shows and sporting events, and for particular events such as the 2002 
Boston Marathon. In addition, tags in the events category can include pre-defined 
calendar events such as New Years Eve, and customized calendar events such as 
birthdays and anniversaries. Tags in the event tag category can contain attributes 
corresponding to the names, locations, and dates of the xmderlying events associated 

25 with the tags. 

The places tag category can be customized to include tag types for particular 
places such as a home, an office, an art museum, or a veication destination. Tags in 
the places tag category can contain attributes corresponding to specific locations that 
are associated with photos, including the name of the location (e.g.. The Metropolitan 
30 Opera House), the names of the city, state, coimtry and region of the world in vrtdch 
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the photos were taken or which are the subject of the photos, and title geographical 
coordinates (e.g., longitude and latitude) for those places. 

Finally, the miscellaneous tag category is as a customizable catchall for tags 
that cannot be easily grouped into a meaningful global category with other tags. 
5 Examples of miscellaneous tag types include tags for an apartment or home search, 
tags for artistic or photos, and tags for particular cars or types of cars. Miscellaneous 
tags can contain attributes corresponding to the name of the subject of the photo, and 
where and when the photo was taken. 

As shown in Fig. 2, the metadata that is associated with a photo can be viewed 
1 0 and edited directly by displaying the photo together with its associated metadata. Fig. 
2 shows a photo entitled "Lori on the road at Legoland" associated with a customized 
people tag, Lori R., and a customized places tag, San Diego. The tags and title 
indicate this is a photo of Lori R. taken on a trip to Legoland in San Diego, CA. This 
photo can be retrieved from the database in any number of different ways, together 
1 5 with different photos tbat are related to this photo in different ways, as discussed 
below. 

In general, photos in the database that have been tagged wiHi one or more tags 
can be searched for and sorted by querying the database for all photos having tags that 
match one or more search tags or the metadata contained within the one or more 

20 search tags. These metadata can include, but are not limited to, data indicating 

whether photos are favorites; frequently viewed; similar to currently selected photos; 
untagged; taken on a particular day or recurring event; shared with or received from 
certain people; imported from certain places; and printed or exported on certain dates. 
In addition, the metadata can include the subject of tiie photo, whether a person, place, 

25 or event; as well as the place and/or event at which the photo was taken. For example, 
the photo of Lori R in Legoland can be retrieved from the database by querying the 
database for all photos tagged with a Lori R. tag. This search will pull up all photos 
of Lori R., including the Legoland photo, regardless of where the photos were taken. 
Alternatively, the Legoland photo can be retrieved by searching the database for all 

30 photos tagged with a San Diego tag. This search will pull up all photos taken in or of 
San Diego, including the Legoland photo, regardless of who is in the photo. Finally, 
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the Legoland photo can be retrieved by searching the database for all photos tagged 
vdUx both a Lori R. tag and a San Diego tag. This search will pull up all photos taken 
in or of San Diego that include Lori including the Legoland photo. 

The database search for photos that match certain tags or groups of tags can be 
5 graphically constructed by dragging various icons representative of tags 350 into a 
graphiced query builder or lens 220, and searching the database for records with 
matching tags or metadata. When search criteria are applied to the photos in the 
databeise, the order in which the photos are displayed is updated so that "best match" 
photos or photos that match all of the search criteria are displayed at the top of an 

10 image area 100 in firont of a first background color or pattern, while "close match" 
photos that match one or more but not all of Hie search criteria are displayed after the 
"best match'' photos and are visually distinguished from them by, for example, being 
displayed in fix>nt of a second backgroimd color or pattern, and "no match" photos that 
fail to match any of the search criteria are displayed at the bottom of the image area in 

1 5 front of a third background color or pattern. 

Perhaps the easiest search to conduct on tagged photos is a search for photos 
taken on a certain date, or within a certain period of time. As previously mentioned, 
among tihe metadata that can be stored with a photo is information indicating the date 
and time a photo was taken. This information is often automatically associated with a 

20 photo when the photo is created or when the photo is scanned into a digital scanner. If 
the photo is created on a digital camera, the camera will generally tag the photo with 
the date and time the photo was taken. If the photo is scamied into a digital scanner, 
the scanner will generally tag the photo with the date and time it was scaimed Iffor 
any reasons neither the digital camera nor digital scanner tags the photo with the date 

25 and time information, flie database will tag the photo with the information when it is 
first imported. 

As shown in Fig. 3, when photos are imported into a database, the temi)oral 
metadata associated vdth the photos can be used to present a histogram of photos in 
the form of a timeline 250 as shown in Fig. 1 . The timeline 250 can show the nimiber 
30 of photos taken as a function of time over some period of time that can range from the 
time the first photo in the database was taken to the present. The timeline 250 can be 
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used by itself, or with other tags 350 to specify the criteria used to search for matching 
photos. The timeline includes adjustable time bands 251 that can be moved to allow 
timeline 250 to specify the time period that is used to find matching photos. 

When the timeline 250 is used by itself to search for matching photos, the 
adjustable time bands 251 can be moved to find all photos in the database that are 
tagged with a date or timestamp that falls within the range indicated by the adjustable 
time bands 25 1 . Photos falling within this range are designated "best match" photos, 
and can be viewed as such in image area 100. For example, the timeline 250 can be 
used by itself to find all photos taken between Jan. 1, 2000 and Feb. 28, 2000 by 
moving the adjustable time bands 251 to these two respective dates. The photos in the 
database that have been tagged with a timestamp &lling between these two dates can 
be retrieved &om the database, and displayed in the "best match" section of image 
area 100. 

In addition to finding photos according to their timestamp, the timeline 250 
can be used with other metadata to limit search tag results. For example, if the 
adjustable time bands 251 of timeline 250 indicate the period of interest extends fi-om 
Jan. 1, 2000 to Feb. 28, 2000, searching the database for all photos having a San 
Diego tag will retum the photo "Lori on the road at Legoland" as a "best match" 
photo, and display the photo in image area 100, only if the photo was taken sometime 
between Jan. 1, 2000 and Feb. 28, 2000. If the photo was taken outside of this time 
period, it would only appear as a "close match" photo in image area 1 00. When tag 
searches are conducted in conjxmction with timeline 250, the timeline displays the 
total number of photos in the database per unit time period in a first color which may 
be a solid color, and the total number of photos in the database that match the tagged 
search criteria as "best" or "close" matches in a second color which may be a hatched 
pattern or color. 

In one embodiment, the timeline 250 shown in Fig. 3 does not display the 
exact number of photos taken during a given period of time, but rather displays a 
vertical bar graph with bar heights that are representative of the number of photos 
taken during a given period of time normalized to the average number of photos taken 
during all such similar periods of time in the database. For example, for a given 
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period of time, the displayed vertical bar can have a height of 0 when no photos have 
been taken during that period; 1 when one to five photos have been taken during that 
period; 2 when the normalized number of photos taken during that period was up to 
50% of the average number of photos taken during all time periods; 3 when the 
5 normalized number of photos taken during that period was between 50% and 80% of 
the average number of photos taken during all time periods; 4 when the normalized 
number of photos taken during that period was between 80% and 120% of the average 
number of photos taken during all time periods; 5 when the normalized nimiber of 
photos taken diocing that period was between 120% and 150% of the average number 
1 0 of photos taken during all time periods; 6 when the normalized mmiber of photos 
taken during that period was between 1 50% and 200% of the avCTage number of 
photos taken during all time periods; and 7 when the normalized number of photos 
taken during that period was more than 200% of the average nimiber of photos taken 
during all time periods. 

15 As shown in Fig. 4, in addition to timeline 250, photos taken on a particular 

day or during a particular month can also be found by displaying the photos in a 2-D 
histogram or scatter plot such as the calendar view shown in the figure. The calendar 
view displays all of the photos that have been taken, scanned, or imported into the 
database on any day in a given month as a function of the day of the week the photos 

20 were taken, and the week in the month. If a particular day of the montii is selected in 
the calendar view, all photos taken on that day can be retrieved from the database as 
"best match" photos. For example. Fig. 3 shows that during the month of June, 2001 
two sets of photos ware taken. The first set contains a single photo taken on June 8, 
while the second set contains 10 photos taken on Jime 18. By selecting the June 18 

25 calendar day, the 10 photos taken on June 18 are selected as the *T>est match" photos, 
and can be displayed ia image area 100. 

The calendar view can also display the restilts of a tag search in the month-at- 
a-glance mode. When so used, each day in the calendar can indicate not only whether 
any photos were taken on that day, but whether the photos taken on that day fail into 
30 the ^'best UMtch", "close match", or "no match" group with respect to the tagged 

search criteria. For example, if the Legoland photo described in Fig. 2 was one of ten 
photos of Lori R. taken in San Diego on June 18, 2001, and a search were done for all 
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photos having a San Diego tag, then the June 18, 2001 square in Fig. 3 would indicate 
that day as having photos in the "best match" group. If, however, a search were done 
for all photos havmg a New York tag, the June 1 8, 2001 square in Fig. 3 would 
indicate that day as having photos in the "no match" group. Finally, if a search were 
done for all photos havmg a New York tag and a Lori R. tag, the June 1 8, 2001 square 
in Fig. 3 would indicate that day as having photos in the "close match" group. 

The particular group into which a set of photos taken on a given calendar day 
falls can be indicated on the calendar using the same color based indication scheme 
used to indicate matching photo groups that are displayed in the viewing area. Thus, 
calendar days containing one or more photos in the "best match" group can be 
presented as v/bitc squares, while calendar days containing one or more photos in the 
"close match" group and no photos in the "best match" group can be presented as blue 
squares, and calendar days containing no photos in either the "best match" or "close 
match" groups can be presented as gray squares. 

As shown in Fig. 5, in addition to the tkneline 250 and calendar views, data 
can be searched for and displayed in an alternate 2-D histogram or scatter plot such as 
a map view. In the map view, the place tag metadata is used to display the geographic 
distribution of photos in the database. Like the timeUne 250 and calendar views, the 
map view can be used to search for photos either by itself, or in conjunction witli one 
or more tag searches. If the map view is used by itself to search for photos, icons 
representing the places where photos in the database have been taken are displayed on 
a map such as the world map shown in Fig. 5. When a location on the map is 
selected, photos taken in that location can be retrieved from the database as photos in 
the **best match" group. For example, if the location of Kenya on the map in Fig. 5 is 
selected, photos taken in Kenya can be selected from the database, and can be 
displayed in image area 100 as "best match" photos. 

The map view can also be tised in conjunction with a tag search. When so 
used, the map view will display not only the distribution of photos as a fimction of 
geographic location, but whether the photos taken at the various geographic locations 
fall in the "best match", "close match", or "no match" group with respect to the tagged 
search criteria. For example, if a search for all photos having an African tag were 
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displayed in tiie map view, the map view would indicate that photos taken from the 
Kenya Safari fall into the "best match" group, while all of the oilier photos shown in 
Fig. 5 fall into the "no match" group. As with the calendar and tunelrne 250 views, 
the particular group iato which a set of photos taken from a given location falls can be 
5 indicated on the map using the same color based indication scheme used to indicate 
matching photo groups that are displayed in image area 100. Thus, locations 
containing one or more photos in the "best match" group can be presented as a white 
area, while locations containing one or more photos in the "close match" group and no 
photos in the '*best match" groirp can be presented as a blue area, and locations 
10 containing no photos in either die '"best match" or "close match" groups can be 
presented as a gray area. 

The map view can be varied in size and shape to accommodate the geographic 
extent of the photos that are either in the database or tibiat are re!q)onsive to a tag search 
conducted on the photos in the database. Thus, if a map view is used to display or 

15 fiirther search among database photos having a North American tag, the map view can 
be limited to a view of the North American continent. The map view can also be 
varied in size and shape by selecting particular regions of a map, such as the western 
region of the United States, or by zooming in and out of the currently displayed map 
region. Photos corresponding to particular locations within a map view, like San 

20 Diego, can be geographically found directly from the map view. 

Once photos matohing a given set of tags are found, they can be selected as a 
group, and various photo management frmctions such as printing, sharing, or 
exporting the photos to a slide show or to a photo album can be performed on the 
group. As a de&ult, all photos in the 'l^est mateh" group are selected when selecting 
25 photos that match a given set of tag search criteria. However, the default can be 
changed such that all photos in both the "close match" and "best match" groups are 
selected when selecting photos that matoh a given set of tag search criteria. 

It should be noted that while the invention has been described in terms of 
managing, cataloguing, searching, and finding digital images and photographs, the 
30 invention can equally be used to manage, catalogue, search for and find other types of 
digital media such as video files, audio files, photo slide shows, and photo albums. 
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These different types of media can be distinguished from one another with a tag of tag 
type media. The media tag, when applied to a media object, can graphically indicate 
the type of media object that is stored in the database. 

As shown in Fig. 6, a video file 600 can be stored in the database and 
5 identified by displaying its first frame together with an overlaid video file icon. 

Similarly, an audio file 610 can be stored in the database and identified by displaying 
the title of the audio file together with an audio file icon. Audio files can be 
associated with and stored as a component part of a slide show or photo albiim, and 
can be played as a soxmdtrack whenever the slide show or photo album is viewed. 
10 Slide shows such as slide show 620, and photo albums such as photo album 630 can 
also be stored in the database, and iconically idratified as shown Fig. 6. Each of these 
objects can be tagged, searched for, and manipulated using the same tools that are 
used to tag, search for, and manipulate digital photos, as previously discussed. 

While the invention has been described as a computer program or algorithm, 

1 5 the invCTtion can be implemented in digital electronic circuitry, or in computer 
hardware, firmware, software, or in any combinations of them. Apparatus of the 
invention can be implemented in a computer program product tangibly embodied in a 
. machine-readable storage device for execution by a programmable processor; and 
method steps of the invention can be performed by a progranmiable processor 

20 executing a program of instructions to perform functions of the invention by operating 
on input data and generating output The invention can be implemented 
advantageously in one or more computer programs fliat are executable on a 
prograrnmable system including at least one programmable processor cotq>led to 
receive data and instructions from, and to transmit data and instructions to, a data 

25 storage system, at least one input device, and at least one output device. Each 

computer program can be implemented in a high-level procedural or object-oriented 
programming language, or in assembly or machine language if desired; and in any 
case, the language can be a compiled or interpreted language. Suitable processors can 
include both general and special purpose microprocessors. Generally, a processor will 

30 receive instructions and data from a read-only memory and/or a random access 

memory. Generally, a computer will include one or more mass storage devices for 
storing data files; such devices include magnetic disks, such as internal hard disks and 
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removable disks; magneto-optical disks; and optical disks. Storage devices suitable 
for tangibly embodying computer program instructions and data include all fonns of 
non-volatile memory, including by way of example semiconductor memory devices, 
such as EPROM, EEPROM, and flash memory devices; magnetic disks such as 
internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks. 
Any of the foregoing can be supplemented by, or incorporated in, application-specific 
integrated circuits (ASICs), 

While the invention has been described in terms of particular embodiments, it 
should be understood that other embodiments are possible as would be ^parent to 
one of ordinary skill in the art. Accordingly, these and other embodiments are within 
the scope of the following claims. 

What is claimed is: 
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1 . A method for graphically constructing a database query, 
comprising: 

receiving a collection of objects, wherein each object in the collection is 
associated with metadata that describes one or more attributes of the object; 

generating a visual representation of a distribution of objects in the collection 
as a function of at least a portion of the descriptive metadata associated with the 
objects; and receiving user input defining a selection in the visxial representation of 
the distribution of objects to construct a database query. 

2. The method of claim 1, wherein the step of generating a visual representation 
of a distribution of objects comprises generating a histogram representing the 
distribution of objects as a function of at least a portion of the descriptive metadata 
associated with the objects, 

3 . The method of claim 2, wherein the step of generating a histogram 
representing the distribution of objects as a function of at least a portion of the 
descriptive metadata associated with the objects comprises generating a timeline 
representing the distribution of objects as a function of temporal metadata associated 
with the objects. 

4. The method of claim 3, wherein the temporal metadata is a date or timestamp 
associated wifli each of the objects. 

5. The method of claim 2, wherein the step of generating a histogram 
representing the distribution of objects as a function of at least a portion of tiie 
descriptive metadata associated with the objects comprises generating a map showing 
the spatial distribution of the objects. 

6. The method of claim 2, wherein the step of generating a histogram 
representing tiie distribution of objects as a function of at least a portion of the 
descriptive metadata associated with the objects comprises generating a calendar 
showing the temporal distribution of the objects. 

7. The method of claim 1, wherein the step of generating a visual representation 
of a distribution of objects as a function of at least a portion of the descriptive 
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metadata associated with the objects comprises generating a scatter plot representing 
the distribution of objects as a function of at least a portion of tiie descriptive metadata 
associated with the objects. 

8. The method of claim 7, wherein the step of generating a scatter plot 
representing the distribution of objects as a function of at least a portion of the 
descriptive metadata associated with the objects comprises generating a map showing 
the spatial distribution of the objects. 

9. The method of claim 7, wherein the step of generating a scatter plot 
representing the distribution of objects as a function of at least a portion of the 
descriptive metadata associated with the objects comprises generating a calendar 
showing the temporal distribution of the objects. 

1 0. The method of claim 1 , wherein the step of receiving user input defining a 
selection of the visual representation of the distribution of objects to construct a 
database query comprises receiving user input selecting a portion of the visual 
representation of the distribution of objects. 

1 1 . The metiiod of claim 1, ^?y^erein the step of receiving user input defining a 
selection of the visual representation of the distribution of objects to construct a 
database query comprises receiving user input limiting the display range of the visual 
representation of the distribution of objects. 

12. A method for managing a collection of objects, 
comprising: 

using one or more search tags to query a database for objects having metadata 
that matches the metadata associated with the one or more search tags; 

categorizing the collection of objects in the database into at least a best match 
groiq) and a no match group according to the results of the search tag query; and 

displaying a representation of the objects firom both the best match group and 
the no match groiq) while distinguishing the objects in the best match groirp from the 
objects in the no match group. 
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13. The method of claim 12, wherein the step of distinguishing the objects in the 
best match group from the objects in the no match group comprises providing a visual 
cue to distinguish the objects in the best match group from the objects in the no match 
group. 

14. The method of claim 13, wherein providing a visual cue to distinguish tlie 
objects in the best match group from the objects in the no match group comprises 
displaying the objects in the best match group in front of a background having a first 
background color, and displaying the objects in the no match group in front of a 
background having a second backgroimd color. 

15. The method of claim 12, fiirther comprising categorizing the objects into a 
close match group; and displaying the objects from the best match group, close match 
group, and no match group while distinguishing the objects in each group from the 
objects in each of the remaining groups. 

1 6. The method of claim 1 2, further comprising generating a visual representation 
of a distribution of the objects in each of the best match and no match groups as a 
function of at least a portion of the descriptive metadata associated with the objects in 
each of the groups. 

17. The method of claim 16, wherein the step of generating a visual representation 
of a distribution of the objects in each of the best match groups and no match groups 
further comprises di^laying a visual representation of a distribution of the objects in 
the best match group and no match group on a histogram. 

1 8. The method of claim 17, wherein the histogram is a timeline representing a 
temporal distribution of the objects in the best match group and no match group. 

1 9. The method of claim 1 7, wherein the histogram is a map showing the spatial 
distribution of the objects in the best match group and no match group. 

20. The method of claim 17, wherein the histogram is a calendar showing the 
temporal distribution of the objects in the best match group and no match group. 
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21 . The. mefhod of claim 1 6, \s4ierein the step of generadhg a visual representation 
of a distribution of the objects in each of the best match groups and no match groups 
further comprises displaying a visual representation of a distribution of the objects in 
the best match group and no match group on a scatter plot 

22. The method of claim 21, wherein the step of generating a scatter plot 
representing the distribution of objects in tiie best match and no match groups 
comprises generating a map shovsdng the spatial distribution of objects in the best 
match and no match groups. 

23 . The method of claim 2 1 , wherein the step of generating a scatty plot 
representing the distribution of objects in the best match and no match groups 
comprises generating a calendar showing the distribution of objects in the best match 
and no match groups as a function of date. 

24. A method for displaying a collection of objects, 
comprising: 

using one or more search tags to query a database for objects having metadata 
that matches the metadata associated with the one or more search tags; and 

displaying a visual representation of a distribution of the objects responsive to 
the search tag query as a function of at least a portion of the metadata. 

25. The method of claim 24, wherein the visual representation of the distribution 
of the objects responsive to the search tag qu^ is a histogram. 

26. The method of claim 25, wiierein the histogram is a timeline representing a 
temporal distribution of the objects responsive to the search tag query. 

27. Thet method of claim 25, wherein the histogram is a map shov^g the spatial 
distribution of the objects in the best match group and no match group. 

28. The method of claim 25, vrfierein the histogram is a calendar showing the 
temporal distribution of the objects in the best match group and no match group. 

29. The method of claim 24, v^erein the visual representation of the distribution 
of the objects responsive to the search tag query is a scatter plot 
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30. The method of claim 29, wherein the scatter plot is a calendar representing the 
distribution of the objects responsive to the search tag query as a function of date. 

3 1 . The method of claim 29, wherein the scatter plot is a map representmg a 
spatial distribution of the objects responsive to the search tag query. 

32. A computer program product for gr^hically constructing a database query, the * 
computer program product comprising instructions operable to cause a programmable 
processor to: 

receive a collection of objects, wherein each object in the collection is 
associated with metadata that describes one or more attributes of the object; 

generate a visual representation of a distribution of objects in the collection as 
a function of at least a portion of the descriptive metadata associated with the objects; 
and receive user input defining a selection in the visual representation of the 
distribution of objects to construct a database query. 

33 . The computer program product of claim 32, wherein the instructions to 
generate a visual representation of a distribution of objects comprise instructions to 
generate a histogram representing the distribution of objects as a function of at least a 
portion of the descriptive metadata associated with the objects. 

34. The computer program product of claim 33, wherein the instmctions to 
generate a histogram representing the distribution of objects as a function of at least a 
portion of the descriptive metadata associated with the objects comprise instmctions 
to generate a timeline representing the distribution of objects as a function of temporal 
metadata associated wdth the objects, 

35. The computer program product of claim 34, wherein tiie temporal metadata is 
a date or timestamp associated with each of the objects. 

36. The computer program product of claim 33, wherein the instructions to 
generate a histogram representing the distribution of objects as a function of at least a 
portion of the descriptive metadata associated with the objects comprise instmctions 
to generate a map showing the spatial distribution of the objects. 



18. 



wo 02/057959 



PCT/US02/01530 



37. The computer program product of claim 33, wherein the instructions to 
generate a histogram representing the distribution of objects as a function of at least a 
portion of the descriptive metadata associated with the objects comprise instructions 
to generate a calendar showing the temporal distribution of the objects. 

3 8. The computer program product of claim 32, whereia the instructions to 
generate a visual representation of a distribution of objects as a function of at least a 
portion of the descriptive metadata associated with the objects comprise instructions 
to generate a scatter plot representing the distribution of objects as a function of at 
least a portion of the descriptive metadata associated with the objects. 

39. The computer program product of claim 38, wherein the instructions to 
generate a scatter plot representing the distribution of objects as a function of at least a 
portion of the descriptive metadata associated with the objects comprise instructions 
to generate a map showing the spatial distribution of the objects. 

40. The computer program product of claim 38, wherein the iostructions to 
generate a scatter plot representing the distribution of objects as a function of at least a 
portion of the descriptive metadata associated with the objects comprise instructions 
to generate a calendar showing the temporal distribution of the objects. 

41 . The computer program product of claim 32, wherein the instmctions to receive 
user input defining a selection of the visual representation of the distribution of 
objects to construct a database query comprise instructions to receive user input 
selecting a portion of the visual representation of the distribution of objects. 

42. The computer program product of claim 32, wherein the instructions to receive 
user input defining a selection of the visual representation of the distribution of 
objects to construct a database query comprise instructions to receive user input 
limiting the display range of the visual representation of the distribution of objects. 

43 . A computer program product for managing a collection of objects, the 
computer program product comprising instructions operable to cause a programmable 
processor to: use one or more search tags to query a database for objects having 
metadata that matches the metadata associated with the one or more search tags; 
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categorize the collection of objects in the database into at least a best match 
group and a no match group according to the resxilts of the search tag query; and 

display a representation of the objects from both the best match group and the 
no match group while distinguishing the objects in the best match group from the 
objects in the no match group. 

44. The computer program product of claim 43, wherein the instructions to 
distinguish the objects in the best match group from the objects in the no match group 
comprise instructions to provide a visual cue to distinguish the objects in the best 
match group from the objects in the no match group. 

45. The computer program product of claim 44, wherein the instructions to 
provide a visual cue to distinguish the objects in the best match group from the objects 
in the no match group comprise instructions to display the objects in the best match 
group in front of a background having a first background color, and to display the 
objects in the no match group in front of a background having a second background 
color. 

46. The computer program product of claim 43, frirther comprising instructions 
operable to cause the progranunable processor to categorize the objects into a close 
match groxq); and to display the objects from the best match group, close match group, 
and no match group while distinguishing the objects in each group from the objects in 
each of the remaining groups. 

47. The computer program product of claim 43, fiarflier comprising instmctions 
operable to cause a programmable processor to generate a visual representation of a 
distribution of the objects in each of the best match and no match groups as a function 
of at least a portion of the descriptive metadata associated with the objects in each of 
the groups. 

48. The computer program product of claim 47, wherein the mstructions to 
generate a visual representation of a distribution of the objects in each of the best 
match groups and no match groups fiirther comprise instructions to display a visual 
representation of a distribution of the objects in the best match group and no match 
gcovcp on a histogram. 
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49. The computer program product of claim 48, wherein the histogram is a 
timeline representing a temporal distribution of the objects in the best match group 
and no match group. 

50. The computer program product of claim 48, wherein the histogram is a map 
showing the spatial distribution of the objects in the best match group and no match 
group. 

5 1 . The computer program product of claim 48, vrfierein the histogram is a 
calendar showing the temporal distribution of the objects in the best match grotq> and 
no match group. 

52. The computer program product of claim 47, wherein the instructions to 
generate a visual representation of a distribution of the objects in each of the best 
match groups and no match grotq>s further comprise instructions to display a visual 
representation of a distribution of the objects in the best match group and no match 
group on a scatter plot. 

53. ITie computer program product of claim 52, wherein the instructions to 
generate a scatter plot representing the distribution of objects in the best match and no 
match groups comprise instmctions to generate a map showing the spatial distribution 
of objects in the best match and no match groups. 

54. The computer program product of claim 52, wherein the instructions to 
generate a scatter plot representing the distribution of objects in the best match and no 
naatch groups comprise instructions to generate a calendar showing the distribution of 
objects in the best match and no match groups as a function of date. 

55. A computer program product for displaying a collection of objects, the 
compiater program product comprising instructions operable to cause a programmable 
processor to: 

use one or more search tags to query a database for objects having metadata 
that matches the metadata associated with the one or more search tags; and 

display a visual representation of a distribution of the objects responsive to the 
search tag query as a function of at least a portion of the metadata. 
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56. The computer program product of claim 55, wherein the visual representation 
of the distribution of the objects responsive to the search tag query is a histogram. 

57. The computer program product of claim 56, wherein the histogram is a 
timeline representing a temporal distribution of the objects responsive to the search 
tag query. 

58. The computer program product of claim 56, wherein tiie histogram is a map 
showing the spatial distribution of the objects in the best match group and no match 
group. 

59. The computCT program product of claim 56, wherein the histogram is a 
calendar showing the temporal distribution of the objects in the best match group and 
no match group. 

60. The computer program product of claim 55, wherein the visual representation 
of the distribution of the objects responsive to the search tag query is a scatter plot. 

61 . The computer program product of claim 60, wherein tiie scatter plot is a 
calendar representing the distribution of flie objects responsive to the search tag query 
as a function of date. 

62. The computer program product of claim 60, wherein the scatter plot is a map 
representing a spatial distribution of the objects responsive to the search tag query. 



22. 



wo 02/057959 



1/6 



PCT/US02/01530 




SUBSTITUTE SHEET (RULE 2&\ 




SUBSTITUTE SHEET (RULE 26) 



wo 02/057959 PCT/US02/01530 

3/6 




SUBSTITUTE SHEET fRULE 



wo 02/057959 PCT/US02/01530 

4/6 




SUBSTITUTE SHEET (RULE 26) 



wo 02/057959 PCT/US02/01530 

5/6 




SUBSTITUTE SHEET fRULE 26) 



wo 02/057959 PCT/US02/01530 

6/6 








SUBSTITUTE SHEET (RULE 26) 



(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 
Iniemational Bureau 

(43) International Publication Date 
25 July 2002 (25.07.2002) 




(HO) International Publication Number 

PCT WO €)2/057959 A3 



(51) International Patent Classification'': G06F 17/30 

(21) International Application Number: PCT/US02/01530 

(22) International Filing Date: 16 January 2002 (16.01.2002) 

(25) Filing language: English 

(26) Publication Language: English 

(30) Priority Data: 

60/261,897 16 January 2001 (16.01.2001) US 

60/179,379 31 January 2001 (31.01.2001) US 

09/774.523 31 January 2001 (31.01.2001) US 

09/997.412 30 November 2001 (30.11.2001) US 

(71) Applicant: ADOIBIE SYSTEMS INCORPORATED 
[US/US]; 345 Paric Avenue, San Jose, CA 95110-2704 

(US). 

(72) Inveators: ROTHMULLER, Kenneth; 4145 Shadow 
Lane #921, Santa Rosa, CA 95405 (US). VERTELNEY, 



Laurie; 2381 Biyant Street. Palo Alto. CA 94301 (US). 
SLATER, Michael; 11224 Occidental Road. Sebastopol. 
CA 95472 (US). PEUTO, Bernard; 1333 Jones Street, 
Apartment 1 106, San Francisco, CA 94109 (US). 

(74) Agent: HORVATH, John, R; Fish & Richardson RC, 500 
Arguello Street #500, Redwood City. CA 94063 (US). 

(81) Designated State (national): JP. 

(84) Designated States (regional): European patent (AT, BE. 
CH. CY. DE. DK. ES, H, PR. GB, GR, IE, IT, LU, MC. 
NL. PT. SE. TR). 

Published: 

— with international search report 

(88) Date of publication of the international search report: 

6 March 2003 

For twO'letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



(54) Title: DIGITAL MEDIA MANAGEMENT APPARATUS AND METHODS 



FINDER 
DRAWER 



< 
m 

e5 



220 \^ 



520 / Mbms S4a 



WiolpMoumol-MfcApl'oPhc lociMeetion \ ~ / /iMtxl 

B« gdtt 3V £b»cI* Ftfld Jog* t«c» \ \ / / 




(57) Abstract: Methods and apparatus for managing, finding and displaying objects such as digital images. Objects are tagged 
("associated") with descriptive textual and numeric data ("metadata"), and stored in a relational database from which they can be 
selected, sorted, and found. Tags can be defined by name, tag type, and associated attributes. Objects can be tagged by dropping a 
tag onto the object, or relating a database record for the tag to a database record for the object. Tagged objects can be searched for and 
displayed according to the degree to which their metadata matches the search criteria. Visual cues can indicate whether displayed 
objects match all, some but not all, or none of the search criteria. Database object distributions can be displayed as histograms or 
scatter plots, including timelines, calendars or maps. Object distributions can be used to search for objects or to limit search results 
for a previous search. 



INTERNATIONAL SEARCH REPORT 



Inten^^j^l Appncation No 

PCT/US 02/01530 



A- CU^SSIRCATlONOFSUBJECT MATTER i. 

IPC 7 606F17/3e 




According to IntemaUonal Patent CtassIflcaUon (IPC) or to both national dasslfication and IPC 




B. FIELDS SEARCHED 


MlntnuimdocumentaDon searched (dassBlcalkm system folkmred tiy dasslfication sym^ 

IPC 7 G06F 


Documentation searctied other than ndnimum documentation to the extent that sudi documents ai 


119 Included in the fields searched 



EPO-Intemal, INSPEC, PAJ 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category' atationofdociniani, with indloaaon, where apprapiiale, of the retevant passages 



Relevant to claim Na 



SHNEIDERMAN, BEN: "Designing the user 

interface: Strategies for effective 

hionan-computer interaction" , ADDISON 

WESLEY , USA XP002210867 

ISBN: 0-201-69497-2 

* Chapter 15: Information Search and 

V1 sualization 

page 519, line 1 -page 521, last line 
page 526, line 3 -page 541, line 8 
figures B2-B6 

-/-- 



1-11, 

24-26, 

29-42, 

55-57, 

60-62 



Further documents are Gstsd fri the oontfrmafion of box C. 



)(1 Patent family memtiers are Dsted In annex. 



^ Special categories dted documents : 

"A' document defining the general state of the art wtiich is not 
considered to l)e of particular relevance 

"E" earlier document but published on or after the international 
fifing date 

■f document which may throw doubts on priority claim(s)or 
which is cited to establish the putilicaDon dale of another 
citation or other special reason (as spedfted) 

•O" document referring to an oral disclosure, use, exhibltton or 
other means 

"P* document pubilshed prior to the international fiOng date but 
later than the prioritjf date dalmed 



"T" later document published after the intemaHonal filing date 
or priority dale and not in conflict with the application but 
dted to urtderstand the principle or theory underlying the 
inventon 

"X" document of parficutar relevance; the claimed invention 
cannot be conddered novel or cannot be considered to • 
Involve an inventive step when the document Is taken alone 

T* document of parfcular relevance; the daimed invention 

cannot be considered to involve an inventive step wfien the 
document is combined with one or mora other such docu- 
ments, such comtdnalion being obvious to a person skiHed 
in the art 

document member of the same patent family 



Dale of the actual completion of the international search 

26 August 2002 



Name and malDng address of the ISA 

European Patent Office, P.B. 5B18 Patentlaan 2 
NL-2280HVRijswuk 
Tel. (+31-70) 340-2040, Tx. 31 651 epo nl, 
Fax: (+31-70)340-3016 



Date of mailing of the international search report 

Oa 12.02 



Au&iortzed officer 



Jaedlcke, M 



Form PCT/ISA/210 (second sheet) (July 1932} 



page 1 of 3 



INTERNATIONAL SEARCH REPORT 



Inlerr^^^^al Application No 

PCT/US 02/Q153O 



C.(ConUnuatlon) DOCUMENTS CONSIDERED TO BE RELEVANT 


Category** 


citation oi document, whh Indteation, wheis appropriate, of t>«e relevant passages 


Relevant to claim No. 


X 


US 5 241 671 A (CARLTON RON ET AL) 
31 August 1993 (1993-08-31) 

column 2, line 30 -column 5, line 6; 
figures 1,16-18,23 

column 18, line 49 -column 20, line 25 


1-11, 

24-26, 
29-42, 
55-57. 
60-62 


X 


US 5 734 888 A (LI SHIH-GONG ET AL) 
31 March 1998 (1998-03-31) 

the whole document 


3,4.6.9, 

26,30, 

34,35, 

37,40, 

57,61 


A 


AHLBERG C: "SPOTFIRE: AN INFORMATION 

EXPLORATION ENVIRONMENT" 

SIGMOD RECORD, ASSOCIATION FOR COMPUTING 

MACHINERY. NEW YORK. US. 

vol. 25,, no. 45, 

1 December 1996 (1996-12-01), pages 25-29, 
XP0907O2066 

page 25, left-hand columns, paragraph 1 
-page 28, right-hand column, paragraph 3; 
figure 1 


5,7.8, 
31,36. 
38,39.62 


A 


BEAZA- YATES, R., RIBEIRO^NETO, B. (EDS.): 
"Modern Infoimnation retrieval" « ADDISON 

1 iwuwi II Aiiivi iiiM ivivii iwvi ■ w V u 1 , nw* «/WI1 

WESLEY , USA XP002210866 
ISBN: 0-2O1-39829-X 

* Chapter 10: User Interfaces and 
Visualization by Marti A. Hearst 
page 257, line 1 -page 339, last line 


1-11, 
24-26 

29-42, 
55-57, 
60-62 


A 


HEARST, MARTI A.: "Next Generation Web 

Search' Settina Our Sites" 

BULLETIN OF THE TECHNICAL COMMITTEE ON 
DATA ENGINEERING, 

vol. 23, no. 3, SepteiTt>er 2000 (2000-09), 
pages 38-48, XPOO2210864 
USA 

the whole document 


1.24,32, 


A 


ROTH S F ET AL: "Toward an information 

visualization workspace: combining 

multiple means of expression" 

HUMAN-COMPUTER INTERACTION, 1997, LAWRENCE 

ERLBAUM ASSOCIATES, USA, 

vol. 12, no. 1-2, pages 131-185, 

XPOO2210865 

ISSN: 0737-0024 

the whole document 

-/- 


1-11, 

24-26, 

29-42. 

55-57. 

60-62 



Form t>CT/ISA/Z10 (eomkniallon of second sheet) (July 1B92) 



page 2 of 3 



INTERNATIONAL SEARCH REPORT 



Inter^^nal Application No 

PCT/US 02/01538 



C^Contlnuatlon) DOCUMENTS CONSIDERED TO BE RELEVANT 



Categoiy « Citation of dociiin«it. with indication, wtiere appropriate, of tlie i«l8vant passages 



Reie>fanttDOIaIni No. 



us 5 844 572 A (SCHOTT ERIC 6) 
1 December 1998 (1998-12-01) 



abstract; figure 19A 

US 5 898 431 A (HOLT BAYLES ET AL) 
27 April 1999 (1999-04-27) 



abstract; figure 4 

US 6 085 205 A (PEAIRS MARK ET AL) 
4 July 2009 (2000-07-04) 



column 3, last line -column 6, line 4; 
figure 5 

PATENT ABSTRACTS OF JAPAN 

vol. 1996, no. 06, 

28 June 1996 (1996-06-28) 

& JP 08 030763 A (CANON INC) . 

2 February 1996 (1996-02-02) 

abstract 

US 6 408 301 Bl (ASHE PHILIP R ET AL) 
18 June 2002 (2002-06-18) 
the whole document 



3,4,6.9, 

26,30, 

34,35. 

37,40. 

57,61 



3,4,6,9. 

26,30. 

34,35. 

37,40, 

57,61 



3.4.6.9. 

26,30. 

34.35. 

37.40. 

57.61 



1.24,32. 
55 



1,24.32. 
55 



Fonn PCT/ISAeiO (DOrtinualan at sKond dieeq (Mr 1999 



uuuL 0 ur 0 



INTERNATIONAL SEARCH REPORT 



lid^Uonal application No. 
PCT/US 02/01530 



Box I Observations where certain claims were found unsearchable (Continuation of Item 1 of first sheet) 



This International Search Report has not been established In respect of certain claims under Article 1 7(2)(a) for the following reasons: 
1 . I I Clainris Nos.: 

because they relate to subject matter not required to be searched by this Authority, namely: 



2. Clabns Nos.: 

because they relate to parts of the International Application that do not comply with the prescribed requlrenients to such 
an extent that no meaningful International Search can be canted out. specifically: 



3. 1^ Claims Nos.: 

because they are dependent claims and are not drafted in accordance with the second and third sentences of Rule 6.4(a), 

Box n Observations where unity of invention is lacking (Continuation of item 2 of first sheet) 

This Intematbnal Searching Authority found multiple Inventions In this International application, as follows: 

see additional sheet 



1 . j I As all required additional search fees were timely paid by the applicant, this International Search Report covers all 
' — * searchable claims. 

2- I As all searchable claims could be searched without effort justifying an additional fee. this Authority did not Invite payment 

of any additional fee. ' 



3, I I As only some off the requiied additional search fees were timely paid by the applicant, this International Search Report 
' — ' covers only those claims for which fees were paid, specifically claims Nos • 



^- LxJ required additional search fees were timely paid by the applicant Consequently, this International Search Report Is 
restricted to the invention first mentioned in the claims; It Is covered by claims Nos.: 

1-11, 24-25, 29-42, 55-57o 60-62 



Remark on Protest 



I I The additional search fees were accompanied by the applicant's protest. 
I I No protest accompanied the payment of additional search fees. 



Fomi PCT/iSA/210 (continuation of first sheet (1)) (July 1 998) 



INTERNATIONAL SEARCH REPORT 



International ^plicaUon No. PCT/US G2/01530 



FURTHER INFORMATION CONTINUED FROM POT/ISA/ 210 



This International Searching Authority found multiple (groups of) 
inventions in this international application, as follows: 

1. Claims: 1-11,24-26,29-42,55-57.60-62 

The first group of claims is directed to a visual 
representation of a distribution of objects as a function of 
metadata associated with these objects and receiving user 
input for visually defining a query using this distribution. 



Z. Claims: 12-23,27,28,43-54,58,59 

This group of claims is directed to the visualization of 
different data categories in a database based on the result 
of a search tag query which selects objects using their 
metadata search tags; in particular, there is a 
categorization of the objects in a best match and a no match 
group. 



INTERNATIONAL SEARCH REPORT 

iinormation on pateni tamny memoers 



Inleri^^^^^al Application No 

PCT/US 02/01530 



Patent document 
cited in search report 


Publication 
date 


Patent family 
meniber(8) 


Publication 
date 


us 5241671 


A 


31-08-1993 


AU 


6908891 A 


31-05-1991 








WU 


yiuoyiD Al 


16-05-1991 








HQ 




00-09-2002 


lie CTOAftftft 

Uo D/o<fooo 


A 
H 


31-03-1998 


HQ 


bbUooyy A 


04-03-1997 








tr 


0627d9Z Al 


07-12-1994 








JP 


6348762 A 


22-12-1994 


us 5844572 


A 


01-12-1998 


US 


5619631 A 


08-04-1997 








AU 


720664 82 


08-06-2000 








AU 


5987796 A 


30-12-1996 








CA 


2196683 Al 


19-12-1996 








EP 


0774135 Al 


21-05-1997 








JP 


10508403 T 


18-08-1998 








NZ 


309656 A 


29-11-1999 








WO 


9641254 Al 


19-12-1996 


US 5898431 


A 




NONE 






US 6085205 


A 


04-07-2000 


CN 


1217509 A 


26-05-1999 








GB 


2331437 A .8 


19-05-1999 








JP 


11224148 A 


17-08-1999 


JP 08030763 


A 


02-02-1996 


US 


6333748 Bl 


25-12-2001 








US 


2002029230 Al 


07-03-2002 


US 6408301 


Bl 


18-06-2002 


NONE 







Form PCT/I8A/210 (patent ramlly annex) (July 1992) 



I 



THIS PAGE BLANK (uspto) 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the apphcant. 

Defects in the images include but are not limited to the items checked: 

□ black borders 

□ image cut off at top, bottom or sides 

□ faped text or drawing 

□ blurred or ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



i 



THIS PAGE BLANK mm) 



